Surgical training apparatus, methods and systems

ABSTRACT

Surgical training apparatus, methods and systems which allow surgical trainees to practice surgical skills on anatomical models in a realistic manner with an augmented reality headset and delivery of targeted surgical coursework curriculum correlated to the actions of the trainee as sensed by sensors in or adjacent the model to help the trainee develop proper surgical technique and decision making.

BACKGROUND OF THE INVENTION

The present invention relates to apparatus, methods and systems forsurgical training. More particularly, the present invention relates tonovel simulated surgical training apparatus, methods and systems whichemploy anatomy models with strategically placed sensors. The sensorequipped models are used in conjunction with an augmented realityheadset and computer software to deliver guided instruction and targetedcurricular content to the trainee at ideal times during a simulatedsurgical training session.

Many surgical trainees currently practice on live patients in theoperating room due to insufficient alternatives. This may lead to lessthan ideal patient outcomes and unnecessarily increased operation times.Studies have shown that longer operating times add to patient risk andincrease the cost of care.

Training on surgical phantom models is known, however, there stillremains a need for improved surgical training apparatus, methods andsystems. Specifically, surgical training requires both visual and verbalcues whilst the student is completing a motor task. Currently, the onlymeans to provide this instruction is through live experiencedpractitioner training session. Unfortunately due to surgeon timeconstraints, in person training is only feasible during actual patientcases and therefore has the potential of causing harm or creatingexcessive costs due to surgical inefficiency.

SUMMARY OF THE INVENTION

Surgical training apparatus, methods and systems which allow surgicaltrainees to practice surgical skills on anatomical models in a realisticmanner with an augmented reality headset and delivery of targetedsurgical coursework curriculum correlated to the actions of the traineeas sensed by sensors in or adjacent the model to help the traineedevelop proper surgical technique and decision making.

The present invention further addresses the above need by providing inanother aspect sensor equipped surgical phantom models with integrateddigital curricular content through an augmented reality headset (AR) orother human computer interface (HCl).

The present invention provides in yet another aspect informationtransfer between surgical phantoms, surgical tools, and computersoftware that allow a user to perform self-guided training.

In yet a further aspect, the present invention provides a surgicaltraining phantom that emits signals in order to prompt delivery ofcurriculum content.

Signals can be generated from models using any suitable electroniccomponents such as, for example, transducers, video images and/or straingauges.

The generation of signals may be initiated in any one of or combinationof ways, such as, for example:

-   -   a) upon sensing a change in the model such as, for example, the        marking of a proposed incision site, the making of an incision,        the onset of simulated bleeding, the resection of a simulated        tumor, etc.;    -   b) upon sensing user movement such as user hand and/or head        motions, for example;    -   c) sensing the use of a surgical instrument such as body marking        pens, suture needles, needle drivers, laparoscopic instruments,        suction tips, etc.; and/or    -   d) sensing a particular video field of view (“FOV”) within the        surgical field or “what the surgeon sees” during the course of        the procedure.

Signals from sensors (e.g., transducers, electromagnetic spectrumemissions including visible and non-visible frequencies) are deliveredto a computer running a surgical training software program using anydesired communication mode such as camera vision, Wi-Fi, Bluetooth,sound, light, wired connection, etc. Machine learning may be employed toparse data, learn from that data and make informed decisions on what ithas learned. Deep learning is a type of machine learning in which amodel learns to perform classification tasks directly from images,texts, or signals. Deep learning may be implemented using neural networkarchitecture which may be computed in real-time by parallel computers.Machine learning and/or Deep learning may be used to identify, processand classify objects using the signals and images from the AR headsetcamera and 9 degree of freedom head/camera position tracker and othersignal outputs from the simulated organs and/or surgical instruments.

Signals are interpreted by the computer surgical training softwareprogram and may cause a state change in the surgical simulation trainingsoftware.

The software may be programmed to deliver tutorial and “how-to” guidesto the trainee that correspond to ongoing progress of the surgicalphantom model training session.

In yet another aspect, the invention provides an AR platform thatdetects a surgical trainee's specific performance during a trainingprocedure on a surgical model and responds to the detected performanceby delivering to the trainee corresponding curricular content and/orother information. The AR headset and/or any video or camera feedincluding, e.g., video from a surgical instrument (laparoscope,endoscope, arthroscope, microscope, etc.), is able to detect one or more“Cues” which may be “Model Cues” and/or “Motion Cues” and/or “StillImage/Video Cues”.

“Model Cues” are discrete elements or physical conditions emanating fromthe model itself which are detectable by a “Cue Receiver” such as the ARheadset. Examples of Model Cues include, but are not limited to,physical markings (e.g., bar codes or other symbols in visible ornonvisible inks), electronic and/or optical sensors and/or any otherfiducials embedded within or applied to the outer surface of thesurgical model.

“Identification (ID) and/or Motion Cues” (hereinafter “ID-Motion Cues”)include detection of physical presence (static state) and/or motions bythe trainee (e.g., eye, head, hand, arm movements) and/or a surgicalinstrument which are detectable by the AR headset. In this regard, thetrainee's body parts and/or the surgical instruments (includingauxiliary items which may be used in the surgery such as clips, spongesand gauze, for example) may be provided with applied (e.g., temporarystick-on) sensors and/or other fiducials that allow detection of thepresence (ID) and/or motion thereof. The motion detection may or may notbe made trackable through a computerized navigation system.

“Still Image/Video Cues” include image capture and video feeds fromsurgical cameras (e.g., laparoscopic, robotic, etc.). The AR headset mayalso have image capture and video feed functionality which creates theinput to the surgical system training software program.

Detection of Model Cues and/or ID-Motion Cues and/or Still Image/VideoCues by the Cue Receiver generates a signal which the surgical trainingsystem software (to which the Cue Receiver is wired or wirelesslyconnected) is programmed to interpret as a specific action and/oranatomical reference point of the model within the context of theparticular surgical training module or session.

The Model Cues are strategically positioned in or on the model in amanner which corresponds with the software programming for thatparticular model. More particularly, the software may be programmed fora particular surgical training module. The software may thus beprogrammed with an ordered (and, optionally, timed) sequence of surgicalacts on the model which are indicative of a successful surgicalprocedure for that particular surgical training module. The types andplacement of the one or more Model Cues in or on the model and/or theID-Motion Cues and/or the Still Image/Video Cues are correlated to theprogrammed ordered sequence of surgical acts for the particular surgicalsession. Should the trainee perform surgical acts on the model that arenot in agreement with the expected surgical performance as identified inthe software program, the software will detect any such digressions andrespond by informing the trainee of the digression from the expectedsurgical protocol.

Curriculum content and/or other information may be automaticallygenerated and delivered to the trainee at the time of the detecteddigression and/or at the conclusion of the training module.

Besides being able to detect a change in the programmed ordered sequenceand/or timing of Model Cue detections, the Model Cues and/or Motion Cuesand/or Image Capture/Video Cues may provide signals to the softwareindicative of a surgical act being performed on the model that is notaccording to protocol for that training module or not meeting thesurgical performance standard for that act (e.g., marking the wrong sitefor an incision on the model with the body marking pen, poorly executingan incision, resection, or improper placement of a surgical instrumentor auxiliary item (such as leaving a sponge in the model).

The system may thus detect the current surgical training state based ona detected Model Cue and/or ID-Motion Cue and/or Image Capture/VideoCues and respond by causing the corresponding curricular content and/orother information to be displayed or otherwise provided to the surgicaltrainee. The software may be programmed with direct visual detectionalgorithms including machine learning, deep learning, and/orreinforcement learning to develop the various Cue detection functions.

In another aspect, the invention provides computer software that isprogrammed to deliver curricular content timed appropriately to thetrainee's progress on surgical training models. The software is based onan algorithm decision tree that selects appropriate content for anygiven surgical training session or scenario. The software structureallows the system to time the delivery of content to the trainee in anydesired manner including immediately after a detected input, if desired.The system may be programmed to include optional playback by the traineeat any interval in the training session.

In another aspect, the invention provides computer software thatsummates the activities detected by the trainee and provides aperformance score for individual steps taken by the trainee and/or theentire procedure of the surgical training module. The output from theCues described above may be summated and interpreted by the machinelearning based on performance differences between novices and experts,for example. The software may also be programmed to calculate aperformance score or provide additional instruction to the trainee inorder to improve future performance.

Additional objects, advantages and novel aspects of the presentinvention will be set forth in part in the description which follows,and will in part become apparent to those in the practice of theinvention, when considered with the attached figures.

DESCRIPTION OF THE DRAWING FIGURES

The above-mentioned and other features and advantages of this invention,and the manner of attaining them, will become apparent and be betterunderstood by reference to the following description of the invention inconjunction with the accompanying drawing, wherein:

FIG. 1A is a perspective view of an embodiment of the invention showingan example of a surgical trainee utilizing an embodiment of the surgicaltraining system;

FIG. 1B is flow chart diagram of an embodiment showing a Cue input,STSS, STSS database and output;

FIG. 1C is a flow chart diagram of an embodiment showing model type andtraining options for each model type including an example of adjacentorgan sensor activation output;

FIG. 2A is a fragmented plan view of an example of surgical phantommodel;

FIG. 2B is the view of FIG. 2A showing markings on the model inaccordance with an embodiment of the invention;

FIG. 3A is a fragmented plan view of the model seen in FIG. 2B andfurther showing a surgical instrument for use by the surgical trainee onthe model in accordance with an embodiment of the invention;

FIG. 3B is the view of FIG. 3A showing the surgical instrument in theprocess of making an incision in accordance with an embodiment of theinvention;

FIG. 3C is the view of FIG. 3B showing simulated bodily fluid (eg;blood, bile, urine, etc.) exiting the incision in accordance with anembodiment of the invention;

FIG. 4A is a fragmented view of a surgical training model having a ModelCue with a trainee holding a surgical instrument above the model inaccordance with an embodiment of the invention;

FIG. 4B is the view of FIG. 4A showing deformation and activation of thesensor applied by the surgical instrument in accordance with anembodiment of the invention;

FIG. 5 is a fragmented view of a surgical model with simulated tumor inthe process of being resected in accordance with an embodiment of theinvention;

FIG. 6A is a fragmented plan view of a surgical model of a human kidneyhaving a simulated tumor in accordance with an embodiment of theinvention;

FIG. 6B is the view of FIG. 6A showing the simulated tumor resected anda temporary clip positioned on the renal artery in accordance with anembodiment of the invention;

FIG. 7 is a flow diagram of a surgical training process in accordancewith an embodiment of the invention;

FIG. 8 is a flow diagram of a surgical training process in accordancewith an embodiment of the invention;

FIG. 9 is a flow diagram of training a computer for machine and deeplearning using neural networks according to an embodiment of theinvention;

FIG. 10 is a flow diagram of real time object/signals detection usingmachine/deep learning according to an embodiment of the invention;

FIGS. 11A and 11B together illustrate a flow diagram of a signaldecision tree and curriculum content delivery according to an embodimentof the invention;

FIG. 12 is a flow diagram of instructional content based on detectedobjects and signals according to an embodiment of the invention;

FIG. 13 is a flow diagram of image timestamp and log according to anembodiment of the invention; and

FIG. 14 is a flow diagram of curriculum instruction playback modeaccording to an embodiment of the invention.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

The surgical training system in its most basic form includes a surgicalmodel, a computer (having the usual computer components including forexample, but not necessarily limited to, a processor, memory,input-output interface, graphic user interface (GUI), etc.), one or more“Cue Receivers” for receiving data inputs in the form of Model Cuesand/or ID-Motion Cues and/or Still Picture/Video Cues, and surgicaltraining system software (“STSS”) running on the computer processor. Thesurgical model may be any organ and/or other anatomical component foundin any animal or human type. The Cue Receiver may include any one orcombination of AR headset, microphone, digital camera, digital video,electronic sensors, real-time clock, touchscreen, computer keyboard,joystick, mouse, trackball, image scanner, graphics tablet, overlaykeyboard, for example. More than one type of Cue Receiver may beprovided on the same device (e.g., AR headset). The Cue Receiver relaysthe received Cue to the STSS which is programmed with one more surgicaltraining sessions or modules. The STSS is programmed to receive andrespond to received Cues generating appropriate output and teaching asurgical trainee to perform a surgical procedure on the surgical model.

Referring to FIG. 1, a surgical trainee 10 is seen wearing an augmentedreality (AR) headset 12 and is operating (i.e., is performing a surgicaltraining session) on a surgical model 14 which may be of any desiredmaterial (e.g., silicone, hydrogel, etc.). The AR headset 12 may includeany one or more but preferably all of the following features or theirequivalents:

1080p DLP Projected Display Waveguide with See through Optics

WiFi & Bluetooth Connectivity 8 Megapixel Camera Quad Core ARM CPU RightEye Monocular Haptic Feedback Voice Control Android 5 OS NoiseCancelling Microphone On Board Video Recording Media

The AR headset 12 may be wired or wirelessly connected to a computerhaving a graphic user interface (“GUI”) 17 which may be in the form of asmart phone running the STSS 19 as a downloaded software application(“app”), for example. The STSS may also be hosted remotely in the“cloud” 21 and provided to a trainee as Software as a Service (SaaS).Any other computer types may be used such as tablets, laptops, desktops, virtual desk top, etc., whereon the STSS may be installed oraccessed as a SaaS. The STSS 19 may be programmed to present to thetrainee a login screen on device 17, monitor 13 and/or AR headset 12wherein the trainee may have a password protected data file which willstore the trainee's surgical training session data for later retrievaland/or playback. The STSS may connect to other servers and/or networkssuch as at 21 b whereby the trainee's STSS file may be connected to thetrainee's personal student data files hosted on, for example, thetrainee's medical school server. As such, the trainee's time spent onsimulated surgical training may be logged for the trainee's class creditor other purposes.

The STSS 19 may be programmed to include one or more of differentsurgical training sessions for selection by the trainee which may bemade by voice command or via the GUI on device 17, for example. Thesurgical training model 14 may include a bar code 14 a or the like whichmay be scanned by a separate bar code scanner 23 a connected to thecomputer 17 through a wired or wireless connection or a scanner 23 bintegral to the computer 17 (e.g., a scanner app of a smart phone 17) orthe STSS app running thereon which is operable to read the bar code onthe model 14 and thereby identify the surgical model anatomy type thetrainee wishes to train on. Each surgical model type programmed into theSTSS may be associated with and displays to the trainee one or moresurgical training sessions which are appropriate to the model type. Forexample, the model type may be identified as a kidney and the matchingsurgical training session options may be presented in a list to thetrainee (e.g., on media interface 17, monitor 13 and/or AR headset 12)as, e.g., (1) tumor resection; (2) kidney stone removal; (3) vesselrupture repair, etc. The trainee may select (input) the desired surgicaltraining session (e.g., by manual input using a graphic user interface(GUI) and touchscreen, keyboard, or mouse, and/or by visual (e.g., eyetracking) and/or voice command) and the STSS is programmed to respond tothe input by launching the trainee's chosen surgical training session ofthe STSS program. Depending on the training session chosen, certainsensor features of the model may be automatically activated by the STSS(but not necessarily triggered) as discussed further below.

As mentioned above, the computer input by a Cue Receiver such as the ARheadset 12 during a surgical training session may include Cues in theform of any one or combination of Model Cues and/or ID-Motion Cuesand/or Still Picture/Video Cues. The various Cue inputs are analyzed bythe STSS as they are received via the Cue Receiver with the STSSresponding by generating output in the form of corresponding curricularcontent and/or other useful information (e.g., alerts of a surgicalemergency being detected, an unsafe procedure being performed, amendedor supplanted protocol to follow due to a deviation from protocol,etc.). The STSS output may be provided in any one or a combination ofdesired formats including audio output and/or display on the AR headset12 and/or on a separate video monitor 13 and/or speaker 15 in theoperating training room. The generated audio output may be provided tothe trainee in the form of alarms and/or verbal instructions, forexample. As such, in this embodiment the trainee receives the generatedoutput for their consideration during the training (real time) so thatthey may understand whether their performance is correct, in generalneed of improvement and/or require they implement a change to thesurgical protocol itself to rectify or address any identified issueswith their performance. The form and content of the generated output maybe programmed into the STSS for each specific surgical training session.The content may be in the form of educational curriculum stored in adatabase 11.

Referring to FIG. 1B, an example of input to STSS and generated outputis seen where the trainee's classroom curriculum content is stored indatabase 11 and is accessible by the STSS 19. In this example, a Cue hasbeen generated and received by the STSS that the trainee made anincision which exceeded the threshold incision depth for the surgicalsession. The STSS is programmed to generate as an output specificcurriculum content from the database 11. In this example, exceeding theincision depth threshold is an act that is tied to Anatomy Lesson 2 andSurgical Technique Lesson 3. These are thus provided as output as seenin box 25 and the trainee can then read these lessons either during orafter the session to improve incision making performance.

FIG. 1C illustrates a box diagram showing the scanned identification ofthe model type where the STSS program may provide multiple surgicalsession choices for each of the model types. The surgical trainingsystem may comprise two or more models with each model being a distinctanatomical model type. The software training program may thus include aseparate surgical session program correlated to each model type. Theseparate surgical session program may further include at least twoalternate surgical session types. A trainee selects (e.g., via interface17) a surgical session and certain ones of the sensors for that modeland surgical session are identified as those that will be utilizedduring that particular training session. For example, if the traineescans a model of a kidney, the STSS presents the list of availablesurgical sessions for the kidney model which in this example are “TumorResection” or “Biopsy”. The trainee selects the model training session(e.g., via interface 17) which causes STSS to launch the program for thechosen session. As mentioned above, the surgical models and sessions mayhave a unique set of sensors that are utilized for the particularsurgical model and session chosen. In the bladder model example, thetrainee may select either the Transurethral Resection or the StoneRemoval training sessions. Selection of the Transurethral Resection mayactivate sensors A and D-F (e.g., power them to a “ready” state via thesensor's firmware, if present) as these sensors are the ones that areassociated with performing that particular surgical session. Should thetrainee instead select the stone removal session, the set of sensorsassociated with that particular surgical session are activated which inthis example are sensors A-C and E. For example, if the chosen model isthe Thoracic model including models of the breast, ribs, lungs andheart, the trainee is presented with the surgical session choices ofBreast Reconstruction and Angioplasty. If the trainee selects BreastReconstruction, the set of sensors associated with that procedure areactivated which include lung sensors D and F and breast sensors K-M.Sensors in adjacent models may be important to indicate if the traineeis wrongly invading the space of the adjacent model structure (e.g.,puncturing a lung while working on the breast). Thus, as seen in box 31of FIG. 1C, the surgical trainee has scanned the Thoracic (model type C)into the STSS which provides the trainee with the choice of performing abreast reconstruction or an angioplasty. The surgical trainee selectsthe breast reconstruction training session at box 33 which launches thatparticular training session program of the STSS. During the surgicaltraining session, should the trainee mistakenly cut into and puncturethe lung, lung sensor D sends a signal to the STSS via a Cue Receiverand the STSS is programmed to respond by providing the trainee with anoutput which may be in the form of a visual warning message in the ARheadset of the lung puncture and need to take remedial action. The STSSmay also provide as an output the curriculum content from database 11corresponding to a lung puncture. The output may be provided in anydesired format and on any media (e.g., on monitor 13, printed on printer9, and/or digitally stored in the memory of computer 17 or on a separatememory storage device such as memory stick 7 for later retrieval by thetrainee, etc.).

An example of a surgical training session is seen in FIGS. 2A and 2Bwherein a surgical phantom model of a human abdomen 16 is seen at thestart of the laparoscopic or robotic surgical training session. Portlocations are designated by reference numerals 18 a-18 d and representthe locations where openings in the abdominal wall (ports) are to beformed by the trainee during the training session. The torso navel orumbilicus is indicated by reference numeral 20 and provides a referencepoint (fiducial) for establishing the general orientation of the modelin 3D space. In this surgical training session example, the trainee isdirected by the STSS via audio and/or visual display in AR headset 12and/or by a separate instruction (which may be in verbal or writtenform, for example) to place a first umbilical port 18 a adjacent navel20. The trainee 10 may further be directed to the proper placement ofumbilical port 18 a by the STSS which may be programmed to cause the ARheadset 12 to superimpose an image of where port 18 a should be placedonto the surgical model 16. Placement of port 18 a causes a Cue (eitherby detecting changes to model appearance, trainee motion, or surface orembedded sensors, for example) to be detected which triggers generationof an input (e.g., electronic signal) to a Cue Receiver such as ARheadset 12. The AR headset 12 (Cue Receiver) relays the signal to theSTSS which is programmed to analyze the input and determine if the inputindicates that the trainee performed proper placement of port 18 a. Ifso, as explained further below, the trainee may then be guided by theSTSS to place the second port 18 b in a location designated by bar code22. Although a bar code 22 is shown in FIG. 2B for the ease ofdescription, it is understood that any type of Model Cue which can besensed by a Cue Receiver (e.g., AR headset 12) may be used which may ormay not be visible to the human eye. Such indicators include, forexample, invisible inks which are detectible by the AR headset 12 butnot the trainee even while wearing the AR headset 12. This may bedesirable as it would require the trainee to find the proper portlocation on the torso without it being immediately recognizable byvisual or other indicators. In this case, the trainee may be simplyinstructed by the STSS to place the second port without being shownwhere that second port is supposed to be placed. The non-visible ModelCue at port location 18 b or other Cue such as the Still Picture/VideoCue may be used to detect if the trainee placed the port in the correctlocation or not.

As stated above, if it is desired to show the trainee the proper portlocation, the STSS may cause either an image of an abdomen with the portin the trainee's AR headset, or overlay an image of the port locationonto the surgical model. The AR headset recognizes the correct locationof the port location 18 b by any desired Cue such as a surface markingsuch as by scanning barcode 22 or other fiducial such as navel 20.Again, this may be as subtle as a slight color change in the model oruse of applied inks or other colorants outside the human visiblespectrum to prevent trainees from relying too heavily on such markingswhich may not be present in actual surgery.

After second port 18 b is correctly placed as detected by theappropriate Cues which relay their received data as input to the STSS(see discussion above), the trainee is directed by the STSS via ARheadset 12 to place a third port 18 c at location 24 which may include aCue in any desired form including, for example, the form of a bar codeon the model as described above or a sensor which may or may not beembedded in the model (and thus not visible to the human eye) that isdetectible by the AR headset 12. The sensor (e.g., a pressure sensor)may, upon activation, generate a signal which is detected by the ARheadset 12 which informs the STSS that third port 18 c has been properlyplaced which is programmed to respond by generating guidance to thetrainee (e.g., by issuing text and/or verbal instructions in AR headset12 and/or monitor 13 and/or speaker 15) to place fourth port at 18 d.After a laparoscopic training procedure is finished as detected by a Cueand relayed to the STSS, the trainee may be instructed by the STSS toremove a specimen from the model (e.g., simulated tumor) and instructthe trainee to create an incision at 26.

Referring now to FIGS. 3A-C, another aspect of the invention may includethe use of image recognition of the trainee's progress to guide thesurgical training session through the STSS. In FIG. 3A, a surgicalinstrument in the form of a scalpel 30 is shown above a surgical modelsurface 32 (trainee's hand not shown in FIGS. 3A-C for the sake ofclarity). The AR headset 12 and STSS may be programmed to receive (e.g.,via AR headset 12) and analyze (process) relative positional data of themodel surface 32, the trainee hand (not shown) and scalpel 30 andprovide the trainee with information (output) as to whether the traineeis holding the scalpel 30 in the correct orientation for the task athand (e.g., creating an incision for tumor resection).

The trainee proceeds with the surgical simulation session by cutting anincision 34 in the model surface 32 using scalpel 30. The AR headset andSTSS may be programmed to calculate the length and/or depth of theincision 34 based on Cues such as visual appearance and/or fiducialreferences. For example, the Cue may be provided in the form of a 1 cmsquare fiducial detected on the model “skin” surface, and wherein theSTSS may be programmed to calculate distance based on the visualdetection of the incision relative to the fiducial. Alternatively, ModelCues in the form of electronic sensors could be spaced a certaindistance apart and the number of sensors detected in linear orcurvilinear sequence can be used for the STSS to calculate distance(length of incision).

Depth (the distance from model surface into body of the model) can beprovided by a Video Cue and/or Motion Cue based on the amount of scalpelthat has extended beneath the upper surface of the model or“disappeared” into the model. The scalpel blade in this case is thevisual cue and is detected by the AR headset 12 which can detect andrelay to the STSS what percentage of the blade has disappeared into themodel. The STSS can be programmed to use this data to calculate incisiondepth and provide appropriate instruction to the trainee if itcalculates that the incision 34 has not been correctly executed, e.g.,it does not meet the minimum programmed thresholds for incision depth.

In FIG. 3C, the model surface 32 may include simulated body fluid suchas simulated blood placed so that it may flow from incision 34. The STSSmay be programmed to recognize this simulated bleeding 36 based ondetected color differentiation (e.g., red, for blood) from model surface32 (e.g., flesh tone), for example, and provide appropriate instructionto the trainee such as “use suction” or “hold pressure”, for example.

FIGS. 4A and 4B illustrate an example of a sensor in the form of apressure sensor 38 which is operable to detect pressures which thesoftware programming correlates into trainee progress and performance.It is noted that while a pressure sensor is used in this example, it isunderstood that any type of sensor may be used including, e.g., strainsensors, flow sensors, etc. For example, the trainee's hand 10 a isholding scalpel 30 (FIG. 4A) which cuts into model surface 32 (FIG. 4B).The sensor is seen in its resting state in FIG. 4A. The sensor 38 may bedetected by the surgical instrument pressing thereagainst as seen inFIG. 4B which generates a signal causing the STSS to provide certaininstructions to the trainee via the AR headset or other HCl (humancomputer interface). When the trainee exerts pressure on sensor 38 viascalpel 30, the sensor 38 sends a signal to a Cue Receiver which relaysthe signal to the STSS which causes the STSS to generate an output inthe form of providing the trainee with the next set of instructions inthe surgical procedure such as, for example, “place self-retainingretractor” in visual and/or audio format. The sensor 38 may also measureand transmit to a Cue receiver and the STSS the force being exertedthereon by the trainee's use of the scalpel. The STSS may compare theforce values received from the sensor 38 and compare the values topreset force thresholds. If the sensed force values are outsideacceptable threshold values, the STSS may respond by generating anoutput of this information to the trainee with the option of furtherinstruction as to how the trainee may correct and/or improveperformance.

FIG. 5 illustrates an example of a surgical model for a surgicaltraining session involving a tumor resection according to an embodimentof the invention. In this example, various sensors are employed to drivethe trainee curriculum content based on sensed trainee performancemetrics. The trainee (not shown) surgical training session on surgicalphantom model 40 directs the trainee to resect tumor 42 with a feedingblood vessel 44. As the trainee resects tumor 42, the blood vessel 44 iscut releasing simulated blood 46. The simulated blood 46 begins to fillthe resection cavity 48 which is sensed by sensors 50 a and 50 b. It isnoted that the sensors may be strategically positioned as shown suchthat they can sequentially detect the amount of blood filling cavity 48.For example, triggering of sensor 50 a but not sensors 50 b or 50 c mayindicate the presence of “n” cubic centimeters (“cc's”) of blood incavity 48. As more blood enters cavity 48, sensor 50 b is triggeredwhich indicates the presence of “n+1” cc's of blood. As even more bloodenters cavity 48 sensor 50 c is triggered indicating “n+2” cc's of bloodin cavity 48. Any number of sensors may be strategically placed in themodel to detect an increase in the amount of blood. Depending on thetype and sensing range of the sensors employed, the sensors may bechosen so as to be activated only upon physical contact with thesimulated blood or they may be activated whenever the simulated blood iswithin a predetermined distance of the sensor warning the trainee thatthe bleeding may be worsening.

The STSS programming may be made such that it selects information toprovide the trainee based on which and/or how many and/or the order ofsensors which are activated during a specific training session or anysegment thereof. In the example shown in FIG. 5, simulated blood 46 hasreached the level of sensor 50 b which may prompt the STSS to provideinstructions to the trainee to better utilize the suction tube 52. Ifthe simulated blood 46 reached the level of sensor 50 c, the STSSprogramming may provide instruction to the trainee, e.g., by providingthe trainee text (e.g., via AR headset 12) and/or voice message such as,for example: “significant blood loss is occurring—alert theanesthesiologist that there is active bleeding.” A sponge 51 is alsoseen having a sensor 53. Should the trainee use sponge 51 and leave itinside the model upon closing the incision, the sensor 53 will signalthe STSS which will alert the trainee that the sponge has been leftinside the model. The STSS may also deliver curriculum content fromdatabase 11 to the trainee correlated to this mistake in procedure.

If there are multiple organs present in the model, sensors within anadjacent organ may be provided to inform the trainee if he/she hasdamaged or wrongly entered the surrounding space of an adjacent organ.For example, as discussed above with reference to FIG. 1C, if thetraining session is on a breast model, lung models may be positioned inan anatomically correct position relative to the breast. Puncturing alung is considered a surgical emergency. The lungs may therefore beprovided with sensors that activate should the trainee knowingly orunknowingly puncture the lung. Upon such sensor activation, the STSS mayissue a text and/or audible alert of the lung puncture with or withoutfurther instruction for the trainee to take corrective action. Ifcorrective action is to be taken, such action may be analyzed (e.g., byvideo and/or other Cues) and timed to determine if the trainee acted inaccordance with STSS programmed accepted surgical corrective actionprotocols.

The STSS programming may instruct the trainee to continue the trainingsession at any time during the session. For example, the programming mayprovide the trainee instructions to use suction tube 52 and forceps 54to retract tumor 42. As the trainee retracts the tumor with the use ofsuction tube 52 and/or forceps 54, a pressure sensor 56 embedded intumor 42 may be pressed upon and thus activated. The STSS programmingmay include threshold pressure values indicative of correct retractionpressure. If insufficient retraction occurs based on low signal fromtumor pressure sensor 56, the STSS may provide an alert to the trainee,e.g., to use suction tube 52 to perform more retraction.

This tumor resection training session may be programmed in the STSS torequire ligation of blood vessel 44 as part of the procedure for tumorremoval. The STSS programming will recognize ligation of vessel whensensor 58 senses a threshold pressure. If a suture is placed aroundvessel 44 but is not sufficiently tight, the STSS programming caninstruct the trainee to redo or tighten the suture to prevent furtherbleeding, for example.

FIGS. 6A and 6B illustrate examples of Model Cues provided in surgicalphantom model 60 intended for kidney tumor resection training. Kidneytumor model 60 is identified by the STSS programming by scanning barcode62 (e.g., with a barcode reader forming a part of AR headset 12 or witha separate barcode scanner such as 23 a or 23 b seen in FIG. 1. Barcode62 may cause the STSS programming to provide the trainee curricularcontent matched to the specific surgical training session (in thisexample, kidney tumor resection). Note barcode 62 is one of manypossible examples of model type marking as a Model Cue. As discussedabove, other examples of sensor detectible markings include subtle colordifferentiation between the marking and the model substrate and/or inksor other colorants outside the human visible spectrum so as to preventthe trainee from noticing the markings which might otherwise create anunrealistic training experience as such markings will not appear in anactual surgical procedure.

Simulated kidney tumor 64 and its border may be identified by the STSSprogramming by sensed color difference between the tumor 64 and thekidney model substrate 60 surrounding the tumor 64. The edge of thekidney model 60 (which in a real kidney is typically covered by fat) hasunique markings that are detected and inform the STSS programming thatthis portion of the kidney has been exposed. The trainee is required toevaluate the entire kidney during surgery to ensure that there are nomissed lesions or other abnormalities. The STSS will instruct thetrainee to “uncover” or “expose” the kidney until marking 66 is detectedby the STSS programming.

During resection of the tumor 64, the trainee must identify the renalartery 68 a and renal vein 68 b. The STSS programming providesinstruction to the trainee to place a temporary clip 70 on only artery68 a. If incorrectly placed (as detected by any one or more of ModelCues and/or ID-Motion Cues and/or Still Image/Video Cues), the STSSprogramming may provide instructions to the trainee that the clip hasbeen improperly placed and/or instruct the trainee to move clip 70 tothe correct position. For example, should the trainee place the clip onthe vein 68 b, this would be detected (e.g., by a sensor placed in or onvein 68 b or by visual input through a camera) and the STSS programmingwould identify it as a medical emergency as placement on the vein wouldcause the kidney to have blood flow in but not out potentially causingthe kidney to burst. Furthermore, the correct clip position isperpendicular to the vessel and the tips of the clip should cross theedge of the vessel. Visual inspection (e.g., color difference betweenclip and vessel) may allow the STSS to assess any overlap and relativepositioning of the clip relative to the artery.

Referring to FIG. 6B, residual tumor is indicated by reference numeral64′. The STSS programming may recognize residual tumor 64′ based onvisual data (e.g., Still Image/Video Cues) such as remaining colordifferentiation from the surrounding area. The STSS programming may theninstruct the trainee to continue with the tumor resection until allresidual tumor 64′ is removed. After residual tumor 64′ is completelyremoved as recognized by a received Cue (e.g., no remaining colordifferentiation seen in the visual data), the STSS programming may theninstruct the trainee to continue with the training session and removetemporary clip 70 to restore blood flow to the kidney.

FIG. 7 illustrates a process for training of the STSS using readilyavailable machine learning software. STSS training is an offline processperformed by taking the Cues of interest in identifying (e.g., suturequality) by using machine learning software for image process training.

In FIG. 8, preprocessed image characteristic data generated as seen inFIG. 7 is used in real time by multiple (typically hundreds) of GPUs(generalized processing units) and/or multiple application specificdigital logic elements within a Field Programmable Gate Array(s)(FPGA(s)), and/or Application Specific Integrated Circuit (ASIC) toprovide a probabilistic estimate of the likelihood that image is aparticular object or image characteristic. This real time imageprocessing approach allows surgical progress to be identified usingCues, interpreted by the STSS, and provide appropriate instruction toprovide training during simulation activity. In this example, the cue isa user's hand and instrument are the cues. Both type of object andtechnique using object are analyzed and then corrected by the computerif deemed necessary.

FIG. 9 is a flow diagram of training a computer for machine and deeplearning using neural networks according to an embodiment of theinvention. Incision spacing is used as an example for detection of usertechnique. The first box represents the input to the cue receiver (astack of images from the AR headset video). A neural network classifiessuture pattern based on a learned image database and determines thespacing distance between sutures. The spacing is compared againstnovice/expert use data and the output is a performance score such aspercentile. The set threshold for acceptable suture spacing and errorthen prompts the STSS to inform user of suture quality andacceptability.

FIG. 10 is a flow diagram of real time object/signals detection usingmachine/deep learning according to an embodiment of the invention;

FIG. 11 illustrates a decision tree depicting “N” instances of theprocess of FIG. 9 running in parallel on multiple GPUs, ASICs and/orFPGAs to identify “Y” objects/signals in the area of interest (AOI)which can be used by the STSS to generate various system output andprompts. Additionally, images can be classified and scored to be usedfor metrics and for figures of merit. For example, attributes such assuture spacing, wound gap or other surgical attributes known to bedesirable for improved surgical outcomes. Figures of merit could beexcellent, good, fair and poor, for example. This processing isperformed iteratively at frame rate “M”, which is normally run at 30-100Hz for simulated images and/or signals. During each frame, objectdetection is performed utilizing machine learning, followed by a processwhich displays/annunciated instructional content based on objectsdetected. Next, classified objects and signals are time stamped andstored in a database for post procedure instructional playback. Lastly,a “time burner” task will run which accounts for unused processing time,and synchronizes the processing at rate of M frames per second.

FIG. 12 illustrates how instructions may be rendered on the AR headset12 to prompt the trainee for the next step in the surgical procedure.Fiducials and/or barcodes are used to estimate the position of detectedobjects within the surgical field of view. Objects detected during thesignal/object detection phase can be overlaid on an image of the area ofinterest and displayed on the AR headset 12 for viewing by the trainee10. For instance, if a surgical trainee is learning suturing, an overlaycan be rendered showing their instantaneous score or figure of merit oftheir suturing technique. For instance, the trainee can provide voiceprompt input, such as “advance instructions to the next step” though theHCl. Additionally, machine learning can also detect whether or not thetrainee is using the correct instrument for a particular phase of thetraining session, and prompt the trainee with the correct instrument.

FIG. 13 illustrates Log image/signal putting a timestamp on detectedobjects and storing them in a database for later retrieval duringplayback/debrief mode.

FIG. 14 illustrates instruction playback mode which retrieves thetraining session using physical simulated organs and time synchronizedAR video/audio/signals from the time-stamped database. Playback can bestarted at any part of the training session in an interactive mannerwith the trainee. For instance, the user can pause the playback at will,jump to a specific time during the training session, jump to the nextinstance of a detected object, or end the playback session. Reports willbe generated at the end of the record/playback session and can includemetrics, scores and a final report record.

While the apparatus, methods and systems of the invention have beenshown and described with reference to certain preferred embodimentsthereof, it will be understood by those skilled in the art that variouschanges in form and details may be made therein without departing fromthe spirit and scope of the invention as described.

What is claimed is:
 1. A surgical training system, comprising: a) amodel of an anatomical part of a human or animal; b) one or more sensorsattached to said model, said one or more sensors operable to emit asignal in response to receiving an activation input; c) an augmentedreality headset having one or more electronic and/or optical input andoutput channels adapted to receive electronic signals from said one ormore sensors; d) a computer processor operable to receive signals fromsaid augmented reality headset and/or said one or more sensors; e) acomputer database having surgical training curriculum having one or moreindividual surgical subject matter components stored therein; and f) asoftware program running on said computer processor and connected tosaid database, said software program operable to correlate said one ormore sensor signals to said one or more individual surgical subjectmatter components, said software program being further operable toprovide as an output the individual surgical subject matter componentwhich is correlated to a received signal.
 2. The surgical trainingsystem of claim 1, wherein said one or more sensors include two or moresensors embedded within said model and exposed upon cutting an openinginto said model, said one or more sensors operable to emit a signal inresponse to contact with a fluid flowing within said opening.
 3. Thesurgical training system of claim 2, wherein two or more sensors arepositioned at different depths within said model and are operable tosense an increasing fluid flow by activation of said sensors insequence.
 4. The surgical training system of claim 3 wherein saidsoftware program is operable to correlate each of said two or moresensors with a volumetric quantity of fluid wherein each of said sensorscorrelates to a different volumetric quantity of fluid.
 5. The surgicaltraining system of claim 4 wherein said software program is operable toprovide an output perceptible to the trainee in response to saidvolumetric quantity of fluid reaching or exceeding a predeterminedthreshold.
 6. The surgical training system of claim 5 wherein saidoutput is one or more of an audible alarm or text message.
 7. Thesurgical training system of claim 6 wherein said output is a textvisible in said augmented reality headset.
 8. A surgical trainingsystem, comprising: a) a model of an anatomical part of a human oranimal, said model having one or more cues; b) a cue receiver operableto receive one or more cues from said model, said cue receiver beingoperable to generate a signal in response to receiving a cue; c) acomputer processor operable to receive signals from said cue receiver;e) a computer database having surgical training curriculum having one ormore individual surgical subject matter components stored therein; andf) a software program running on said computer processor and connectedto said database, said software program operable to correlate said oneor more received signals to said one or more individual surgical subjectmatter components, said software program being further operable toprovide as an output the individual surgical subject matter componentwhich is correlated to a received signal.
 9. The surgical trainingsystem of claim 8, wherein said cue receiver is an augmented realityheadset having one or more of the following: a) projected displaywaveguide with see through optics; b) WiFi and Bluetooth connectivity;c) camera; d) central processing unit; e) monocular; f) binocular; g)haptic feedback; h) voice control; i) operating system; j) noisecancelling microphone; and k) on board video recording media.
 10. Thesurgical training system of claim 8 wherein said cue receiver is adigital still and/or video camera.
 11. The surgical training system ofclaim 8 wherein said cue is a barcode attached to said model and saidcue receiver is a bar code scanner.
 12. The surgical training system ofclaim 8 wherein said software program is provided as software as aservice.
 13. The surgical training system of claim 8 wherein said outputis provided on a computer monitor.
 14. The surgical training system ofclaim 9 wherein said output is provided on said augmented realityheadset.
 15. The surgical training system of claim 8 and furthercomprising two or more of said models with each model being a distinctanatomical model type, and wherein said software program includes aseparate surgical session program correlated to each of said modeltypes.
 16. The surgical training system of claim 15 wherein saidseparate surgical session program includes at least two alternatesurgical session types.
 17. The surgical training system of claim 16wherein said anatomical part is thoracic having heart and lung models.18. The surgical training system of claim 8 wherein said anatomical partincludes at least two models and wherein said software program isprogrammed to provide an output in response to signals received fromeither of said models.
 19. The surgical training system of claim 16wherein said model type is kidney and said surgical session typesinclude a tumor resection surgical session and a biopsy session.
 20. Asurgical training system, comprising: a) a model of an anatomical partof a human or animal; b) one or more sensors attached to said model,said one or more sensors operable to emit a signal in response toreceiving an activation input; c) a computer processor operable toreceive signals from said one or more sensors; e) a computer databasehaving surgical training curriculum having one or more individualsurgical subject matter components stored therein; and f) a softwareprogram running on said computer processor and connected to saiddatabase, said software program operable to correlate said one or morereceived sensor signals to said one or more individual surgical subjectmatter components, said software program being further operable toprovide as an output the individual surgical subject matter componentwhich is correlated to a received signal.
 21. The surgical trainingsystem of claim 20 wherein said sensor is a proximity sensor and saidactivation input is provided by a surgical instrument positionedadjacent said sensor.
 22. A method for performing a simulated surgicaltraining session on a simulated anatomical model of an animal or human,said method comprising the steps of: a) providing a simulated anatomicalmodel of an animal or human; b) providing a computer connected to adatabase with educational curriculum correlated to said simulatedanatomical model; c) providing one or more cue receivers operable tosend signals to said computer; d) providing software on said computer,said software programmed with one or more surgical training sessionmodules correlated to said anatomical model, said software programmed toaccess said database and provide as a human perceptible output saideducational curriculum in response to receiving a signal from said cuereceiver.